Trends in Obstetrics and Gynecology Residency Applications in the Year After Abortion Access Changes.

Importance: State-specific abortion restrictions currently affect the training of approximately 44% of obstetrics and gynecology (OBGYN) residents in the US. Examination of where future trainees apply for residency is important. Objective: To assess changes in the percentage of applicants to OBGYN residency programs by state based on abortion restrictions in place after the Dobbs v Jackson Women's Health Organization (hereafter, Dobbs v Jackson) US Supreme Court decision and examine whether applicants' preference for programs, as suggested by the distribution of application signals that express higher interest, was associated with abortion bans. Design, Setting, and Participants: This serial cross-sectional study used anonymized data for all applicants to OBGYN residency programs in the US during September and October from 2019 to 2023. Data were obtained from the Association of American Medical Colleges Electronic Residency Application Service. Exposures: Applications and program preference signals sent to OBGYN residency programs, analyzed by applicants' self-reported demographics. Main Outcomes and Measures: The primary outcome was differences in the percentage of unique applicants to OBGYN residency programs from 2019 to 2023, with programs categorized by state-based abortion restrictions after the Dobbs v Jackson decision. Secondary outcomes included the distribution of program signals by state abortion ban status. Results: A total of 2463 applicants (2104 [85.4%] women) who applied to OBGYN programs for the 2023 residency match cycle were the focal sample of this study. While overall applicant numbers remained stable between 2019 and 2023, the number of applicants differed significantly by state abortion ban status in the 2022 (F2,1087 = 10.82; P < .001) and the 2023 (F2,1087 = 14.31; P < .001) match cycles. There were no differences in the number of signals received by programs in states with bans after controlling for known covariates such as number of applications received and program size, and there were no differences in the percentage of signals sent by out-of-state applicants to programs in states with different abortion laws than their home states (F2,268 = 2.41; P = .09). Conclusions and Relevance: In this cross-sectional study, there was a small but statistically significant decrease in the number of applicants to OBGYN residency programs in states with abortion bans in 2023 compared with 2022. However, applicant signaling data did not vary by states' abortion ban status. While OBGYN residency programs almost completely filled in 2023, continued monitoring for the potential consequences of state abortion bans for OBGYN training is needed.

Predicting Medical Student Performance With a Situational Judgment Test in Admissions.

PURPOSE: To explore correlations between AAMC situational judgment test (SJT) scores, other admissions data, and learners' medical school performance. METHOD: First- and second-year medical students from 8 U.S. MD-granting medical schools completed a prototype version of the AAMC SJT in 2017. Outcomes included research-only faculty ratings of student performance, final course grades, and faculty evaluations of student performance, 2017-2018 and 2018-2019 academic years. Bivariate correlations were used to investigate the relationship between SJT scores and student performance outcomes and hierarchical regressions to investigate whether SJT scores provided incremental validity over MCAT total scores and cumulative undergraduate grade point averages (UGPAs) for predicting student performance outcomes. RESULTS: In general, there were small positive correlations with research-only faculty ratings from the first year of medical school, with the highest for social skills/service orientation ( rcorrected = .33, P < .05). Correlations were higher, with the highest for social skills/service orientation and cultural competence ( rcorrected = .33 and .36, respectively, P < .05) in the second year in medical school. SJT scores improved prediction of research-only faculty ratings over MCAT total scores and UGPAs for reliability and dependability/capacity for improvement, cultural competence, social skills/service orientation, and the overall composite score in the first year and for resilience and adaptability, social skills/service orientation, cultural competence, and the overall composite score in the second year. SJT scores demonstrated small correlations with course grades ( rsample-weighted = .10, P = ns) and faculty evaluations related to professionalism skills ( rsample-weighted = .14, P < .05); however, MCAT total scores explained most of the variance associated with course outcomes. CONCLUSIONS: These studies provide initial evidence that SJT scores may add value to the medical school admissions process because scores were related to faculty ratings of professional behaviors and provided unique information relative to MCAT scores and UGPAs.

Designing a Situational Judgment Test for Use in Medical School Admissions.

ABSTRACT: It has long been acknowledged that professional competencies are required for success in medical school, residency training, and medical practice. Over the last decade, medical schools have begun to introduce standardized assessments of professional competencies, but many still rely on interviews to assess these competencies, which occur after about half of the applicant pool has already been screened out. In this article, the authors discuss the development, evaluation, and launch of the Association of American Medical Colleges (AAMC) situational judgment test (SJT) for use in medical school admissions. The AAMC SJT is designed to assess an examinee's understanding of effective and ineffective behaviors related to the core competencies for entering medical students, including service orientation, social skills, cultural competence, teamwork, ethical responsibility to self and others, reliability and dependability, resilience and adaptability, and capacity for improvement. The authors evaluate the evidence for the need for SJTs in medical school admissions by exploring common derailers in medical school, gaps in the admissions process regarding information about professional competencies, and the challenge of conducting holistic review in a high-volume context. They summarize existing research from the employment, international medical education, and residency selection contexts suggesting that SJT scores are positively associated with subsequent performance and may add value to the admissions process. The authors discuss 5 goals that were the foundation for developing the AAMC SJT: (1) assess the professional competencies needed for success in medical school using a proven method, (2) enable holistic review in a high-volume admissions context, (3) create and share a program of research to support the appropriate use of SJT scores, (4) signal the need for preparation in professionalism to learners, and (5) balance the need for a new assessment with minimizing the burden and risk for applicants.

Impact of Preference Signals on Interview Selection Across Multiple Residency Specialties and Programs.

Background Program signaling is an innovation that allows applicants to express interest in specific programs while providing programs the opportunity to review genuinely interested applicants during the interview selection process. Objective To examine the influence of program signaling on "selected to interview" status across specialties in the 2022 Electronic Residency Application Service (ERAS) application cycle. Methods Dermatology, general surgery-categorical (GS), and internal medicine-categorical (IM-C) programs that participated in the signaling section of the 2022 supplemental ERAS application (SuppApp) were included. Applicant signal data was collected from SuppApp, applicant self-reported characteristics collected from the MyERAS Application for Residency Applicants, and 2020 program characteristics collected from the 2020 GME Track Survey. Applicant probability of being selected for interview was analyzed using logistic regression, determined by the selected to interview status in the ERAS Program Director's WorkStation. Results Dermatology had a 62% participation rate (73 of 117 programs), GS a 75% participation rate (174 of 232 programs), and IM-C an 86% participation rate (309 of 361 programs). In all 3 specialties examined, on average, signaling increased the likelihood of being selected to interview compared to applicants who did not signal. This finding held across gender and underrepresented in medicine (UIM) groups in all 3 specialties, across applicant types (MDs, DOs, international medical graduates) for GS and IM-C, and after controlling for United States Medical Licensing Examination Step 1 scores. Conclusions Although there was variability by program, signaling increased likelihood of being selected for interview without negatively affecting any specific gender or UIM group.

Self-Reported Disadvantage in Medical School Admissions: A Call to Review, Revise, and Further Advance Holistic Review.

PURPOSE: This study examined how applicants interpret the self-reported disadvantaged (SRD) question in the American Medical College Application Service (AMCAS) application. METHOD: Data from 129,262 applicants who applied through AMCAS from 2017 through 2019 were used, including financial and family history, demographic characteristics, and work status and residence. Fifteen applicants from the 2020 and 2021 AMCAS cycles were interviewed about their experiences with the SRD question. RESULTS: Large effects were found for SRD applicants with fee assistance waivers ( h = 0.89), Pell grants ( h = 1.21), state or federal aid ( h = 1.10), and parents with less education ( h = 0.98) and non-SRD applicants with a large proportion of their education paid by family ( d = 1.03). Another large difference was found for reported family income distribution (73% of SRD applicants reporting family income < $50,000 vs 15% of non-SRD applicants). More SRD applicants were Black or Hispanic (26% vs 16% and 5% vs 5%), Deferred Action for Childhood Arrivals recipients (11% vs 2%), born outside the United States (32% vs 16%), and raised in a medically underserved area (60% vs 14%). There was a moderate effect for first-generation to college SRD applicants ( h = 0.61). SRD applicants had lower Medical College Admission Test scores ( d = 0.62) and overall and science grade point averages ( d = 0.50 and 0.49, respectively) but no meaningful differences in acceptance or matriculation rates. The interviews identified 5 themes: (1) unclear disadvantage definition; (2) different perceptions of disadvantage and overcoming challenges or obstacles ; (3) identification as disadvantaged or not; (4) SRD essay content; and (5) concerns about lack of transparency in how the SRD question is used in admissions. CONCLUSIONS: Revising the SRD question by including context, phrasing, and instructions for broader experience categories might be beneficial because of lack of transparency and understanding.

Interview Invitations for Otolaryngology Residency Positions Across Demographic Groups Following Implementation of Preference Signaling.

Importance: Preference signaling is a new initiative in the residency application process that has been adopted by 17 specialties that include more than 80% of applicants in the 2023 National Resident Matching cycle. The association of signals with interview selection rate across applicant demographics has not been fully examined. Objective: To assess the validity of survey-based data on the association of preference signals with interview offers and describe the variation across demographic groups. Design, Setting, and Participants: This cross-sectional study examined the interview selection outcomes across demographic groups for applications with and without signals in the 2021 Otolaryngology National Resident Matching cycle. Data were obtained from a post-hoc collaboration between the Association of American Medical Colleges and the Otolaryngology Program Directors Organization evaluating the first preference signaling program used in residency application. Participants included otolaryngology residency applicants in the 2021 application cycle. Data were analyzed from June to July 2022. Exposures: Applicants were provided the option of submitting 5 signals to otolaryngology residency programs to indicate specific interest. Signals were used by programs when selecting candidates to interview. Main Outcomes and Measures: The main outcome of interest was the association of signaling with interview selection. A series of logistic regression analyses were conducted at the individual program level. Each program within the 3 program cohorts (overall, gender, and URM status) was evaluated with 2 models. Results: Of 636 otolaryngology applicants, 548 (86%) participated in preference signaling, including 337 men (61%) and 85 applicants (16%) who identified as underrepresented in medicine, including American Indian or Alaska Native; Black or African American; Hispanic, Latino, or of Spanish origin; or Native Hawaiian or other Pacific Islander. The median interview selection rate for applications with a signal (48% [95% CI, 27%-68%]) was significantly higher than for applications without a signal (10% [95% CI, 7%-13%]). No difference was observed in median interview selection rates with or without signals when comparing male (46% [95% CI, 24%-71%] vs 7% [95% CI, 5%-12%]) and female (50% [95% CI, 20%-80%] vs 12% [95% CI, 8%-18%]) applicants or when comparing applicants who identified as URM (53% [95% CI, 16%-88%] vs 15% [95% CI, 8%-26%]) with those who did not identify as URM (49% [95% CI, 32%-68%] vs 8% [95% CI, 5%-12%]). Conclusions and Relevance: In this cross-sectional study of otolaryngology residency applicants, preference signaling was associated with an increased likelihood of applicants being selected for interview by signaled programs. This correlation was robust and present across the demographic categories of gender and self-identification as URM. Future research should explore the associations of signaling across a broad range of specialties and the associations of signals with inclusion and position on rank order lists and match outcomes.

The AAMC Standardized Video Interview: Lessons Learned From the Residency Selection Process.

Calls to change the residency selection process have increased in recent years, with many focusing on the need for holistic review and alternatives to academic metrics. One aspect of applicant performance to consider in holistic review is proficiency in behavioral competencies. The Association of American Medical Colleges (AAMC) developed the AAMC Standardized Video Interview (SVI), an online, asynchronous video interview that assesses applicants' knowledge of professionalism and their interpersonal and communication skills. The AAMC worked with the emergency medicine community to pilot the SVI. Data from 4 years of research (Electronic Residency Application Service [ERAS] 2017-2020 cycles) show the SVI is a reliable, valid assessment of these behavioral competencies. It provides information not available in the ERAS application packet, and it does not disadvantage individuals or groups. Yet despite the SVI's psychometric properties, the AAMC elected not to renew or expand the pilot in residency selection.In this Invited Commentary, the authors share lessons learned from the AAMC SVI project about introducing a new tool for use in residency selection. They recommend that future projects endeavoring to find ways to support holistic review engage all stakeholders from the start; communicate the value of the new tool early and often; make direct comparisons with existing tools; give new tools time and space to succeed; strike a balance between early adopters and broad participation; help stakeholders understand the limitations of what a tool can do; and set clear expectations about both stakeholder input and pricing. They encourage the medical education community to learn from the SVI project and to consider future partnerships with the AAMC or other specialty organizations to develop new tools and approaches that prioritize the community's needs. Finding solutions to the challenges facing residency selection should be a priority for all stakeholders.

Adding Long-term Value to the Residency Selection and Assessment Process.

The residency selection process is challenging for both applicants and programs. This is particularly true in competitive specialties such as otolaryngology. The importance of noncognitive competencies in successful residents has been well demonstrated in both medicine and surgery. Unfortunately, there is no streamlined or uniform process for incorporating this information into the selection and training of residents. This review provides a summary of innovative approaches in the selection of residents in otolaryngology and the important role these methods and associated data can play in resident selection and training. The goal of these processes is to create a holistic view of potential residents so programs can enhance the development of current residents and boost long-term success in our specialty. Laryngoscope, 130:65-68, 2020.

The AAMC Standardized Video Interview and the Electronic Standardized Letter of Evaluation in Emergency Medicine: A Comparison of Performance Characteristics.

PURPOSE: To compare the performance characteristics of the electronic Standardized Letter of Evaluation (eSLOE), a widely used structured assessment of emergency medicine (EM) residency applicants, and the Association of American Medical Colleges (AAMC) Standardized Video Interview (SVI), a new tool designed by the AAMC to assess interpersonal and communication skills and professionalism knowledge. METHOD: The authors matched EM residency applicants with valid SVI total scores and completed eSLOEs in the 2018 Match application cycle. They examined correlations and group differences for both tools, United States Medical Licensing Examination (USMLE) Step exam scores, and honor society memberships. RESULTS: The matched sample included 2,884 applicants. SVI score and eSLOE global assessment ratings demonstrated small positive correlations approaching r = 0.20. eSLOE ratings had higher correlations with measures of academic ability (USMLE scores, academic honor society membership) than did SVI scores. Group differences were minimal for the SVI, with scores slightly favoring women (d = -0.21) and US-MD applicants (d = 0.23-0.42). Group differences in eSLOE ratings were small, favoring women over men (approaching d = -0.20) and white applicants over black applicants (approaching d = 0.40). CONCLUSIONS: Small positive correlations between SVI score and eSLOE global assessment ratings, alongside varying correlations with academic ability indicators, suggest these are complementary tools. Findings suggest the eSLOE is subject to similar sources and degrees of bias as other common assessments; these group differences were not observed with the SVI. Further examination of both tools is necessary to understand their ability to predict clinical performance.

Applicant Reactions to the AAMC Standardized Video Interview During the 2018 Application Cycle.

PURPOSE: This study examined applicant reactions to the Association of American Medical Colleges Standardized Video Interview (SVI) during its first year of operational use in emergency medicine (EM) residency program selection to identify strategies to improve applicants' SVI experience and attitudes. METHOD: Individuals who self-classified as EM applicants applying in the Electronic Residency Application Service 2018 cycle and who completed the SVI in summer 2017 were invited to participate in 2 surveys. Survey 1, which focused on procedural issues, was administered immediately after SVI completion. Survey 2, which focused on applicants' SVI experience, was administered in fall 2017, after SVI scores were released. RESULTS: The response rates for surveys 1 and 2 were 82.3% (2,906/3,532) and 58.7% (2,074/3,532), respectively. Applicant reactions varied by aspect of the SVI studied and their SVI total scores. Most applicants were satisfied with most procedural aspects of the SVI, but most applicants were not satisfied with the SVI overall or with their total SVI scores. About 20% to 30% of applicants had neutral opinions about most aspects of the SVI. Negative reactions to the SVI were stronger for applicants who scored lower on the SVI. CONCLUSIONS: Applicants had generally negative reactions to the SVI. Most were skeptical of its ability to assess the target competencies and its potential to add value to the selection process. Applicant acceptance and appreciation of the SVI will be critical to the SVI's acceptance by the graduate medical education community.

Innovation in Residency Selection: The AAMC Standardized Video Interview.

PURPOSE: Innovative tools are needed to shift residency selection toward a more holistic process that balances academic achievement with other competencies important for success in residency. The authors evaluated the feasibility of the AAMC Standardized Video Interview (SVI) and evidence of the validity of SVI total scores. METHOD: The SVI, developed by the Association of American Medical Colleges, consists of six questions designed to assess applicants' interpersonal and communication skills and knowledge of professionalism. Study 1 was conducted in 2016 for research purposes. Study 2 was an operational pilot administration in 2017; SVI data were available for use in residency selection by emergency medicine programs for the 2018 application cycle. Descriptive statistics, correlations, and standardized mean differences were used to examine data. RESULTS: Study 1 included 855 applicants; Study 2 included 3,532 applicants. SVI total scores were relatively normally distributed. There were small correlations between SVI total scores and United States Medical Licensing Examination Step exam scores, Alpha Omega Alpha Honor Medical Society membership, and Gold Humanism Honor Society membership. There were no-to-small group differences in SVI total scores by gender and race/ethnicity, and small-to-medium differences by applicant type. CONCLUSIONS: Findings provide initial evidence of the validity of SVI total scores and suggest that these scores provide different information than academic metrics. Use of the SVI, as part of a holistic screening process, may help program directors widen the pool of applicants invited to in-person interviews and may signal that programs value interpersonal and communication skills and professionalism.

How good are we at selecting students that meet our mission? Outcomes of the 2011 and 2012 entering classes selected by a locally developed multiple mini interview.

BACKGROUND: Can a locally developed multiple mini interview (MMI) process lead to outcomes reflective of local values and mission? METHODS: In 2017, the authors performed a retrospective analysis of the relationship of MMI with multiple-choice-based outcomes and non-multiple-choice-based outcomes, including clerkship competencies, OSCE, scholarship/service/leadership, academic honor society induction, peer and faculty humanism nominations, and overall performance at graduation for two entering classes with acceptance decisions based exclusively on a locally developed MMI. RESULTS: There was no association between MMI and performance on multiple-choice-based examinations. For other outcomes, the effect size of MMI for OSCE was small and leadership/service and scholarship did not correlate with MMI score. For clerkship competencies, there was medium effect size for patient care, practice-based learning and improvement, interpersonal and communication skills, and cultural competence. Highest and lowest quartile MMI scorers were no different in academic honor society induction; however, top quartile MMI scorers received more humanism votes versus last quartile and were more likely rated outstanding or excellent graduates. CONCLUSIONS: Local development of MMI and of admissions processes with sole reliance on MMI for final acceptance decisions will not affect academic preparation/medical school performance in multiple-choice-based assessments but can lead to locally desired attributes in students.

The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculants.

PURPOSE: Most research examining the predictive validity of the Medical College Admission Test (MCAT) has focused on the relationship between MCAT scores and scores on the United States Medical Licensing Examination Step exams. This study examined whether MCAT scores predict students' unimpeded progress toward graduation (UP), which the authors defined as not withdrawing or being dismissed for academic reasons, graduating within five years of matriculation, and passing the Step 1, Step 2 Clinical Knowledge, and Step 2 Clinical Skills exams on the first attempt. METHOD: Students who matriculated during 2001-2004 at 119 U.S. medical schools were included in the analyses. Logistic regression analyses were used to estimate the relationships between UP and MCAT total scores alone, undergraduate grade point averages (UGPAs) alone, and UGPAs and MCAT total scores together. All analyses were conducted at the school level and were considered together to evaluate relationships across schools. RESULTS: The majority of matriculants experienced UP. Together, UGPAs and MCAT total scores predicted UP well. MCAT total scores alone were a better predictor than UGPAs alone. Relationships were similar across schools; however, there was more variability across schools in the relationship between UP and UGPAs than between UP and MCAT total scores. CONCLUSIONS: The combination of UGPAs and MCAT total scores performs well as a predictor of UP. Both UGPAs and MCAT total scores are strong predictors of academic performance in medical school through graduation, not just the first two years. Further, these relationships generalize across medical schools.

An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s?

PURPOSE: To investigate current medical school admission processes and whether they differ from those in 1986 when they were last reviewed by the Association of American Medical Colleges (AAMC). METHOD: In spring 2008, admission deans from all MD-granting U.S. and Canadian medical schools using the Medical College Admission Test (MCAT) were invited to complete an online survey that asked participants to describe their institution's admission process and to report the use and rate the importance of applicant data in making decisions at each stage. RESULTS: The 120 responding admission officers reported using a variety of data to make decisions. Most indicated using interviews to assess applicants' personal characteristics. Compared with 1986, there was an increase in the emphasis placed on academic data during pre-interview screening. While GPA data were among the most important data in decision making at all stages in 1986, data use and importance varied by the stage of the process in 2008: MCAT scores and undergraduate GPAs were rated as the most important data for deciding whom to invite to submit secondary applications and interview, whereas interview recommendations and letters of recommendation were rated as the most important data in deciding whom to accept. CONCLUSIONS: This study underscores the complexity of the medical school admission process and suggests increased use of a holistic approach that considers the whole applicant when making admission decisions. Findings will inform AAMC initiatives focused on transforming admission processes.

Core personal competencies important to entering students' success in medical school: what are they and how could they be assessed early in the admission process?

Assessing applicants' personal competencies in the admission process has proven difficult because there is not an agreed-on set of personal competencies for entering medical students. In addition, there are questions about the measurement properties and costs of currently available assessment tools. The Association of American Medical College's Innovation Lab Working Group (ILWG) and Admissions Initiative therefore engaged in a multistep, multiyear process to identify personal competencies important to entering students' success in medical school as well as ways to measure them early in the admission process. To identify core personal competencies, they conducted literature reviews, surveyed U.S and Canadian medical school admission officers, and solicited input from the admission community. To identify tools with the potential to provide data in time for pre-interview screening, they reviewed the higher education and employment literature and evaluated tools' psychometric properties, group differences, risk of coaching/faking, likely applicant and admission officer reactions, costs, and scalability. This process resulted in a list of nine core personal competencies rated by stakeholders as very or extremely important for entering medical students: ethical responsibility to self and others; reliability and dependability; service orientation; social skills; capacity for improvement; resilience and adaptability; cultural competence; oral communication; and teamwork. The ILWG's research suggests that some tools hold promise for assessing personal competencies, but the authors caution that none are perfect for all situations. They recommend that multiple tools be used to evaluate information about applicants' personal competencies in deciding whom to interview.

Validity of four approaches of using repeaters' MCAT scores in medical school admissions to predict USMLE Step 1 total scores.

BACKGROUND: This study investigated the validity of four approaches (the average, most recent, highest-within-administration, and highest-across-administration approaches) of using repeaters' Medical College Admission Test (MCAT) scores to predict Step 1 scores. METHOD: Using the differential predication method, this study investigated the magnitude of differences in the expected Step 1 total scores between MCAT nonrepeaters and three repeater groups (two-time, three-time, and four-time test takers) for the four scoring approaches. RESULTS: For the average score approach, matriculants with the same MCAT average are expected to achieve similar Step 1 total scores regardless of whether the individual attempted the MCAT exam one or multiple times. For the other three approaches, repeaters are expected to achieve lower Step 1 scores than nonrepeaters; for a given MCAT score, as the number of attempts increases, the expected Step 1 decreases. The effect was strongest for the highest-across-administration approach, followed by the highest-within-administration approach, and then the most recent approach. CONCLUSIONS: Using the average score is the best approach for considering repeaters' MCAT scores in medical school admission decisions.